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Abstract — Randomized (dithered) quantization is a method ca- 
pable of achieving white reconstruction error independent of the 
source. Dithered quantizers have traditionally been considered 
within their natural setting of uniform quantization. In this 
paper we extend conventional dithered quantization to nonuni- 
form quantization, via a subterfage: dithering is performed 
in the companded domain. Closed form necessary conditions 
for optimality of the compressor and expander mappings are 
derived for both fixed and variable rate randomized quantization. 
Numerically, mappings are optimized by iteratively imposing 
these necessary conditions. The framework is extended to in- 
clude an explicit constraint that deterministic or randomized 
quantizers yield reconstruction error that is uncorrelated with 
the source. Surprising theoretical results show direct and simple 
connection between the optimal constrained quantizers and their 
unconstrained counterparts. Numerical results for the Gaussian 
source provide strong evidence that the proposed constrained 
randomized quantizer outperforms the conventional dithered 
quantizer, as well as the constrained deterministic quantizer. 
Moreover, the proposed constrained quantizer renders the re- 
construction error nearly white. In the second part of the 
paper, we investigate whether uncorrelated reconstruction error 
requires random coding to achieve asymptotic optimality. We 
show that for a Gaussian source, the optimal vector quantizer 
of asymptotically high dimension whose quantization error is 
uncorrelated with the source, is indeed random. Thus, random 
encoding in this setting of rate-distortion theory, is not merely a 
tool to characterize performance bounds, but a required property 
of quantizers that approach such bounds. 

Index Terms — Source coding, dithered quantization, subtrac- 
tive dithering, compander, quantizer design, analog mappings. 



I. Introduction 

Dithered quantization is a randomized quantization method 
introduced in JTI. A central motivation for dithered quantiza- 
tion is its ability to yield quantization error that is independent 
of the source, which can be achieved if certain conditions, 
determined by Schuchman, are met j2). Traditionally dithered 
quantization has been studied in the framework where the 
quantizer is uniform (with step size A) and the dither signal is 
uniformly distributed over (— ~, ~), matched to the quantizer 
interval as shown in Figure 1. A uniformly distributed dither 
signal is added before quantization and the same dither signal 
is subtracted from the quantized value at the decoder side. 
Note that only subnactive dithering is considered in this paper. 
In the variable rate case, the quantized values are entropy 
coded, conditioned on the dither signal. Randomized (dithered) 
quantizers have been studied in the past due to important 
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properties that differentiate them from deterministic quantiz- 
ers, and were employed to characterize rate-distortion bounds 
for universal compression (3j, (4j. Zamir and Feder provide 
extensive studies of the properties of dithered quantizers (5), 

Beyond its theoretical significance, randomized quantiza- 
tion is of practical interest. Many filter/system optimization 
problems in practical compression settings, such as the "rate- 
distortion optimal filterbank design" problem |7), or low rate 
filter optimization for DPCM compression of Gaussian auto- 
regressive processes J8), assume quantization noise that is 
independent of (or uncorrrelated with) the source. Although 
this assumption is satisfied at asymptotically high rates |9), 
such systems are mostly useful for very low rate applications. 
For example, in (8), it is stated that the assumptions made 
in the paper are not satisfied by deterministic quantizers, 
and that dithered quantizers satisfy the assumptions exactly. 
However, conventional (uniform) dithered quantization suffers 
from suboptimal compression performance. Hence, a quantizer 
that mostly satisfies the assumptions, but at minimal cost in 
performance degradation, would have considerable impact on 
many such applications. 

In this paper, we consider a generalization to enable ef- 
fective dithering of nonuniform quantizers. To the best of 
our knowledge, this paper is the first attempt (other than our 
preliminary work in fTO] , JTTJ) to consider dithered quantiza- 
tion in a nonuniform quantization framework. One immediate 
problem with nonuniform dithered quantization is how to 
apply dithering to unequal quantization intervals. In traditional 
dithered quantization, the dither signal is matched to the 
uniform quantization interval while maintaining independence 
of the source, but it is not clear how to match the generic dither 
to varying quantization intervals. As a remedy to this problem, 
we propose dithering in the companded domain. We derive 
the closed form necessary conditions for optimality of the 
compressor and expander mappings for both fixed and variable 
rate randomized quantization. We numerically optimize the 
mappings by iteratively imposing these necessary conditions. 

However, the resulting (unconstrained randomized) quan- 
tizer does not render reconstruction error orthogonal to the 
source. Therefore, we extend the framework to include an ex- 
plicit such constraint. Surprising theoretical results show direct 
and simple connections between the optimally constrained ran- 
dom quantizers and their unconstrained counterparts. We note 
in passing that the nonuniform dithered quantizer subsumes the 
conventional uniform dithered quantizer as an extreme special 
case. 

For the variable rate case, the proposed nonuniform dithered 
quantizer is expected to outperform the conventional dithered 
quantizer, most significantly at low rates where the optimal 
variable rate (entropy coded) quantizer is often far from 
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Fig. 1. The basic structure of dithered quantization 



uniform. We observe that a deterministic quantizer cannot 
render the quantization noise independent of the source but can 
make it uncorrelated with the source. We hence also present an 
alternative deterministic quantizer that provides quantization 
noise uncorrelated with the source. We derive the optimality 
conditions of such constrained quantizers, for both fixed and 
variable rate quantization, and compare their rate-distortion 
performance to that of randomized quantizers. 

Dithered quantization offers an interesting theoretical twist. 
Randomized quantization is an instance of the random en- 
coding principle used to elegantly prove the achievability of 
coding bounds in rate distortion theory fT2) . However, to 
actually achieve those bounds, a random encoding scheme 
is not necessary, as they can be approached by a sequence 
of deterministic quantizers of increasing block length. In the 
second part of the paper, we investigate the settings under 
which randomized quantization is asymptotically necessary. A 
trivial example involves requiring source-independent quan- 
tization error. It is obvious that the reconstruction (hence 
quantization error) is a deterministic function of the source 
when the quantizer is deterministic Q, while conventional 
dithered quantization produces quantization error that is in- 
dependent of the source. Although a deterministic quantizer 
can never render the quantization error independent of the 
source, it can produce quantization error uncon-elated with the 
source. A natural question is whether the rate distortion bound, 
subject to the uncorrelated error constraint, can be achieved 
(asymptotically) with a deterministic quantizer. 

The paper is organized as follows: In Section III, we 
present the proposed nonuniform randomized quantizers, along 
with its extension to constrained randomized quantizer that 
renders the quantization error orthogonal to the source. In 
Section IV, we derive the necessary conditions of optimality 
for the deterministic quantizer that generates reconstruction 
error uncorrelated with the source. In Section V, we study 
the asymptotic (in quantizer dimension) results, and show that 
for a Gaussian source optimal constrained quantizer must be 
randomized. Experimental results that compare the proposed 
quantizers to the conventional dithered quantizer are presented 
in Section VI. We discuss the obtained results and summarize 
the contributions in Section VII. 

II. Review of Dithered Quantization 

A. Notation and Preliminaries 

In general, lowercase letters (e.g., x) denote scalars, bold- 
face lowercase (e.g., x) vectors, uppercase (e.g., U, X) ma- 



trices and random variables, and boldface uppercase (e.g., 
X) random vectors. E[-], Rx, and Rxz denote the expec- 
tation, covariance of X and cross covariance of X and Z 
respectively V denotes the gradient. jV(/i, K) denotes the 
Gaussian random vector with mean and covariance matrix 
K. 

The entropy of a discrete random vector source X £ M. K 
taking values in X is 

H(X) = -J2P(X = x)logP(X = x) (1) 

xex 

where logarithm is base 2 to measure it in bits. The differential 
entropy of a continuous random variable X with probability 
density function fx(x) is 

h(X) = - J f x (x) log f x (x)dx (2) 

The divergence between two densities fx and gx, is given by 

Wx\\9x)= [ fx(x)\og f ^\dx (3) 
J 9x(x) 

The divergence definition above can be extended to conditional 
densities. For joint densities, Jxy an d 9xy the conditional di- 
vergence T>(Jx\y\\9x\y) is defined as the divergence between 
the conditional distributions fx\Y an d 9x\y averaged over the 
density f Y (y): 

T>(fx\Y\\9x\Y)= ffviv) f Jx\y{x, y) log ^ X]Y ^' V \ dxdy 
J J 9x\Y{x,y) 

(4) 

The mutual information between two random variables X 
and Y with marginal densities fx(x) and fy{y) an d a joint 
density fxY(x,y) is given by 

I{X,Y)= [ [ f x , Y {x,y)\og rfsdy (5) 

J J fx{x)f Y {y) 

Zero-mean vectors x £ M. K and y € K M are said to be 
uncorrelated if they are orthogonal: 



E[yx T ] = 

where the right hand size is M x if matrix of zeros. 



(6) 



' We assume zero mean random variables. This assumption is not necessary, 
but it considerably simplifies the notation. Therefore, it is kept throughout the 
paper. 
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Fig. 2. The proposed nonuniform dithered quantizer 



B. Dithered Quantization 

A quantizer is denned by a set of reconstruction points 
and a partition. The partition V = {Pi} associated with a 
quantizer is a collection of disjoint regions whose union covers 
M. K . The reconstruction points 1Z = {r^} are typically chosen 
to minimize a distortion measure. The vector quantizer is a 
mapping Qk ■ R K — > R K that maps every vector X g ~R K 
into the reconstruction point that is associated with the cell 
containing X, i.e. 

Q K {x) = n if XeVi (7) 

While our theoretical results are general, for a vector quan- 
tizer of arbitrary dimensions, for presentation simplicity, we 
will primarily focus on scalar quantization in the treatment of 
numerical optimization of nonuniform dithered quantizer and 
for experimental results. The nonuniform dithered quantization 
approach is directly extendable to vector quantization by 
replacing the companded domain uniform quantizer with a 
lattice quantizer, although at the cost of significantly more 
challenging numerical optimization. 

The scalar uniform quantizer, with reconstructions 
{0, ±A, ±2A, ±TA}, is a mapping Q : R ->• K such that 

Q(x)=iA for iA- A/2 < x < iA + A/2 (8) 

In fixed rate quantization, the range parameter T is determined 
by the rate Rf 

R f =\og(2T + l) (9) 

while in variable rate quantization T need not, in principle, 
be finite and we will assume T — > oo. In this case, uniform 
quantization is followed by lossless source encoding (entropy 
coder). 

Let dither Z be a random variable, distributed uniformly 
on the interval (—A/2, A/2). Then, conventional dithered 
quantizer approximates the source X by 

X = Q(X + Z)-Z (10) 

It can be shown that the reconstruction error of this quan- 
tizer (denoted N) is independent of the source value X — x, 
i.e., N = X - X = Q(X + Z) - Z - X is independent 
of X and uniformly distributed over (— A/2, A/2) for all X. 
Contrast that with a deterministic quantizer, whose error is 
completely determined by the source value |9|. 

We note that for this property to hold, the quantizer should 
span the support of the source density i.e., there should 
be no overload distortion. While this is often the case for 
variable rate quantization, for fixed rate overload distortion 
is inevitable if the source has unbounded support such as a 



Gaussian source. For practical purposes though, it is common 
to assume that the source has finite support and we also 
follow this assumption in our analysis of fixed rate randomized 
quantization: the quantization error of conventional (uniform) 
dithered quantization is assumed to be independent of the 
source. 

The realization of the dither random variable Z is available 
to both the encoder and the decoder. Thus, assuming an 
optimal entropy coder, the rate of the variable rate quantizer 
tend to the conditional entropy of the reconstruction given the 
dither, i.e., 

R v = H{X\Z) = H{Q{X + Z)\Z) (11) 
In (5), it was shown that the following holds: 

H(Q{X + Z)\Z) = h(X + N)~ log A (12) 

We will use (jT2j in the rate calculations of the variable rate 
(entropy coded) randomized quantization. 

III. Nonuniform Dithered Quantizer 

The main idea is to circumvent the main difficulty due to 
unequal quantization intervals by performing uniform dithered 
quantization in the companded domain (see Figure 2). The 
source X is transformed through compressor g(-) before 
undergoing dithered uniform quantization. At the decoder 
side, the dither is subtracted to obtain Y. Since we perform 
uniform dithered quantization in the companded domain, it is 
easy to show that Y = g(X) + N, where N is uniformly 
distributed over (— A/2, A/2) and independent of the source. 
The reconstruction is obtained by applying the expander 
X = w(Y). The objective is to find the optimal compressor 
and expander mappings g(-),w(-) that minimize the expected 
distortion under the rate constraint. The MSE distortion can 
be written as: 

D = j f[x- w(g(x) + n)] 2 f x (x)f N (n)dxdn (13) 

where /W(n) is uniform over (— A/2, A/2). Interestingly, this 
problem bears some similarity to the joint source channel 
mapping problem where the optimal analog encoding and 
decoding mappings are studied fPJ) . In our setting, the quan- 
tization error is analogous to the channel noise and the rate 
constraint in variable rate quantization plays a role similar 
to that of the power constraint. Similar to (T3J, we develop 
an iterative procedure that enforces the necessary conditions 
for optimality of the mappings. Note that the conventional 
(uniform) dithered quantizer is a special case employing the 
trivial identity mappings, i.e., g(x)—w(x)=x,Vx. 
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A. Optimal Expander 

The conditional expectation w(y) = E{X\Y = y} minimizes 
MSE between the source and the estimate. Hence, the optimal 
expander w is 

7+ 

/ xf x (x)dx 



w(y) 



7+ 

/ fx(x)dx 

7- 



(14) 



where for fixed rate 7+ = min{g 1 (AT), g 1 (y + A/2)} and 
7_ = max{5 _1 (— AT), g^ 1 {y — A/2)}, while for variable 
rate 7+ = g^ 1 (y + A/2) and 7_ = g~ 1 (y - A/2). 

Note: We restrict the discussion to regular quantizers 
throughout this paper, hence g(-) is monotonically increasing. 

B. Optimal Compressor 

Unlike the expander, the optimal compressor cannot be writ- 
ten in closed form. However, a necessary optimality condition 
can be obtained by setting the functional derivative of the cost 
to zero. Thus, a locally optimal compressor g(-), for a given 
expander w(-), requires that the functional derivative of the 
total cost, J, along the direction of any variation function r](-) 
vanishes p4*| , i.e., 

d 

J [g(x) + en(x)] = 0, Vx e R (15) 



VJ 



ck 



=0 



for all admissible perturbation functions rj(-). 

1) Fixed rate: For fixed rate, we have granular distortion, 
denoted D g , and overload distortion, denoted D \. Note that 
we must account for the overload distortion here, as this 
constrains g(x) from growing unboundedly in the iterations 
of the proposed algorithm. Since the rate is fixed, the total 
cost is identical to the distortion, i.e., Jf = D g + D Q i where 
D g and D [ are: 



A/2 g-\AT) 



D„ 



A 



[x — w(g(x) + n)] 2 fx(x)dxdn (16) 



-A/2g-i(-AT) 



A/2 ( 9 -'(-AT) 



A 



[x - w(-TA + n)ff x {x)dx 



-A/2 



[x - w(TA + n)} 2 f x (x)dx 



g~ 1 (AT) 



> dn 



(17) 



2) Variable rate: The rate is obtained via (12) and ([12}, 
which require the distribution of Y = g(X) + N: 

Mv) = i [Fx(g-\y + A/2))- F x {g-\y- A/2))] 

(18) 

where Fx(x) is the cumulative distribution function of X. 
The rate is then evaluated as 



The total cost for variable rate quantization is J v = D + XR 
where A is the Lagrangian parameter that is adjusted to obtain 
the desired rate. 

C. Design Algorithm 

The basic idea is to iteratively alternate between enforcing 
the individual necessary conditions for optimality, thereby 
successively decreasing the total cost. Iterations are performed 
until the algorithm reaches a stationary point. Solving for 
the optimal expander is straightforward since the expander 
is expressed in closed form as a functional of the known 
quantities, g(-), fx(-)- Since the compressor condition is not 
in closed form, we perform steepest descent, i.e., move in 
the direction of the functional derivative of the total cost with 
respect to the compressor mapping g. By design, the total cost 
decreases monotonically as the algorithm proceeds iteratively. 



The compressor mapping is updated according to ( 20 1, where 
i is the iteration index, V J [<?(•)] is the directional derivative 
and /i is the step size. 



g i+ i{x) = gi{x) - fiVJ[g] 



(20) 



R v = h(Y)- log A 



(19) 



Note that there is no guarantee that an iterative descent 
algorithm of this type will converge to the globally optimal 
solution. The algorithm will converge to a local minimum 
and hence, initial conditions are important in such greedy 
optimizations. A low complexity approach to mitigate the poor 
local minima problem, is to embed within the solution the 
"noisy channel relaxation" method of ]T5) , fl6) . We initialize 
the compressor mapping with random initial conditions and 
run the algorithm for a very low rate (large value for the 
Lagrangian parameter A). Then, we gradually increase the 
rate (decrease A) while tracking the minimum. Note that local 
minima problem is more pronounced at multi-dimensional op- 
timizations, which hence requires more powerful non-convex 
optimization tools such as deterministic annealing fTT) . In 
our design and experiments, we focus on scalar compressor 
and expander and we did not observe significant local minima 
problems. 

IV. Reconstruction Error Uncorrelated with the 
Source 

In this section, we propose two quantization schemes (one 
deterministic, one randomized) that satisfy the constraint that 
reconstruction error be uncorrelated with the source. 

A. Constrained Deterministic Quantizer 

A deterministic quantizer cannot yield quantization noise 
independent of the source [9|. However, it is possible to render 
the quantization noise uncorrelated with the source. An early 
prior work along this line appeared in JTS), where a uniform 
quantizer is converted to a quantizer whose quantization noise 
is uncorrelated with the source, by adjusting the reconstruction 
points. In this section, we derive the optimal (nonuniform in 
general) deterministic quantizer which is constrained to give 
quantization error uncorrelated with the source. 

Let 77 and 77 be the reconstruction points and Vi and V L 
represent the i th quantization region, for the constrained (i.e., 
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whose quantization error is uncorrelated with the source) and 
unconstrained MSE optimal quantizer, respectively. Also, let 
Pi and pi denote the probability of X falling into the i th cell 
of these respective quantizers. 



Theorem 1 Vi = Vi and r 7 - 



Cvi , Vi where 

-l 



C = R X (E^n 



Proof: We start with the fixed rate analysis. Let M 
denote the number of quantization cells. The distortion can 
be expressed as 

M „ 

^ = E I (x-r l ) T (x-r l )f x {x)dx (21) 
l=1 xev % 

and the "uncorrelatedness" constraint may be stated via the 
orthogonality principle 



M . 

^2 / x ( x - r i) T fx (x) dx = 



Note further that |22) can be written as: 

M 



^ rrff = R x where U= J xf x (x) dx 



(22) 



(23) 



xeVt 



The constrained problem of minimizing D subject to 



M 



Y^, fili = Rx is equivalent to the unconstrained minimiza- 

i=i 

tion of Lagrangian J: 



K 



J = D + J2l(kf 



fc=i 



M 



Rx(k)-J2 r Mk) 



(24) 



where 7 = [7(1) 7(2)... , j(K)] denotes the KxK multiplier 
Lagrangian matrix, Rx(k) denotes the k th column of Rx and 
li(k) denotes the k th element of Zj. By setting V ri J = 0, we 
obtain the condition: 



A" 



-21 



2 Pi rJ -J2^(kfh(k) =0 (25) 



k=l 



Noting that Y,k=i"l( k ) T h{k) = iff, we obtain n = jrCU 
where C is a constant M x M matrix. C is found by plugging 
this into d23l: 



(26) 



Note that Zj/p, is the MSE optimal reconstruction of an un- 
constrained quantizer that shares the same decision boundary 



with the constrained one, Vi. Plugging (26 1 into (21 1 and after 
some algebraic manipulations, we obtain: 



D 



'x 



'X 



D* 



D* 



(27) 



where D* is the distortion associated with the quantizer given 
by Vi and with corresponding optimal reconstruction points 



li/pi- (27 1 implies that D achieves its minimum whenever 
D* is minimized. Hence, 



Vi = Vi =*> U = pifi 



(28) 

Plugging ( f28] > into ( [26] ), we obtain the result. The proof 
for variable rate follows similar lines, with the only mod- 
ification that we now have to account for the rate term 



R 



^Pilogpi. The uncorrelatedness constraint is identical 

i=l 

to the one in fixed rate, hence the overall Lagrangian cost can 
be expressed as: 



J = D 



M 

E 

fc=i 



l(kf 



Rx(k) 



M 

E 

<=i 



rih(k) 



M 

E^ 

=1 



+ A > >ilogpi 



(29) 



By setting V ri J 



and following the same steps, we obtain: 

-1 




I, 



(30) 



Note that ( |27| > holds due to ( |30[ > and the optimal unconstrained 
quantizer achieves the minimum distortion D* subject to 
the rate constraint. This indicates that the constrained and 
unconstrained quantizers have identical pi and hence V% = Vi 



which implies Z s ; = pifi. Plugging Zj = p^i into (30i, we 
obtain the desired result. 



B. Constrained Randomized Quantizer 

Due to the effect of companding, the nonuniform random- 
ized quantizer we derived in Section III does not guaran- 
tee reconstruction error uncorrelated with the source even 
though it builds on the (conventional) dithered quantizer whose 
quantization error is independent of the source. We therefore 
explicitly constrain the randomized quantizer to generate un- 
correlated reconstruction error, by adding a penalty term to 
the total cost function. The Lagrangian parameter A c > is 
set to ensure E{xw(g(x) + n)} = E{x 2 }. 

J c = J + \ c E[x 2 - xw(g{x) + n)] (31) 

where J = J v in the case of variable rate and J = Jf 
for fixed rate. We find the necessary conditions of optimality 
of constrained compressor and expander mappings at fixed 
and variable rate, by setting the functional derivative of the 
total cost ( J c ) to zero. Surprisingly, the optimally constrained 
compressor mapping remains unchanged (compared to the 
unconstrained optimal compressor) and the only modification 
of the optimally constrained expander mapping is simple 
scaling. We state this result in the following theorem. 

Theorem 2 Let g and w be the compressor and expander 
mappings of the unconstrained optimal randomized quantizer. 
Let g c and w c denote the optimal mappings subject to the 
constraint that the reconstruction error be uncorrelated with 
the source. Then, 

9c{x) = g(x),w c (y) = (1 - \ c )w(y) (32) 

where A c is the Lagrangian multiplier of 07). 
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Note that this result applies to both fixed and variable rate. 

Proof: The optimal expander is no longer the standard 
conditional expectation, since it is impacted by the constraint. 
By setting 

£ J c [w(y) + er)(y)] = (33) 

we obtain the optimal expander in closed form as w c (y) = 
(1 — X c )w(y). The update rule for g c (x) can be derived 
similarly. Setting 



de 



J c [g{x) + er)(x)] = 



(34) 



e=0 



and plugging w c (y) = (1 — X c )w(y) yields, after straightfor- 
ward algebra, g c {x) = g(x). ■ 

V. Asymptotic Analysis 

A. Rate-Distortion Functions 

To quantify theoretically how much a sourc^] can be 
compressed under the independent/uncorrelated reconstruction 
error constraint, we define two rate-distortion functions in 
which we respectively constrain the reconstructions error to 
be i) uncorrelated with the source Rjj{D), and ii) independent 
of the source Rj(D). 

Assume that we have source X with density fx(-) that pro- 
duces the independent identically distributed (i.i.d.) sequence 
X\,X<2,..,X n denoted as X n . Similarly, let X\, X2, .., X n 
be the reconstruction sequence, denoted as X n . Let S n = 
X n — X n be the i.i.d. sequence of reconstruction errors with 
marginal density fs(-)- Let fxs(x, s) denote joint distribution 
of X and S and d(X n ,X n ) denote the distortion measure 
between sequences X n and X n defined as 

n 

d(X n ,X n ) = -y2d(X i ,X i ) (35) 
n * — ' 
i=i 

Let us recall the classical rate-distortion result in information 
theory: 

Rate-distortion Theorem: (eg. [ 121) Let R{D) be the in- 
fimum of all achievable rates R with average distortion 
E[d(X n ,X n + S n )] < D as n 00. Then, 

R(D) = inf I(X; X + S) (36) 

S:E[d(X,X+S)]<D 

We next focus on our problem: let Rjj(D) be the infimum 
of all achievable rates R with average distortion 



lim E[d(X n ,X n + S n )} < D 

n— foo 

subject to the constraint 

E[XiSi] =Q,i = l,2,..,n 



(37) 



(38) 



as n — > 00. Similarly, let Ri(D) be the infimum of all achiev- 
able rates R with average distortion E[d(X n , X n + S n )} < D 
subject to the constraint Si is independent of X^ for all i, 

2 The notation in this section is limited to scalar sources for simplicity, it 
is trivial to extend the results to vector sources albeit with more complicated 



as n — > 00. Then, we have the following result characteriz- 
ing the fundamental limits of source compression under the 
constraints that reconstruction error is uncorrelated with or 
independent of the source. 



Theorem 3 



Ru(D)= inf I(X;X + S) (39) 

S:E[d(X,X+S)]<D 
E[XS]=0 



RAD) = inf 

S:E[d(X,X+S)]<D 
V(f xs (X,S)\\fx(X)fs(SJ)=0 



I{X; X + S) (40) 



Proof: Consider the distortion measures 

djj(x, x + s) = d(x, x + s) + 0\\xs\\ (41) 

d!(x, x + s)= d(x, x + s) + plog / X . S [ X ; 8 ) (42) 

fx{x)fs{s) 

for some /3 > 0. 

We next consider the rate-distortion functions (denoted 
Ry(D) and Rj(D)) associated with these distortion measures. 
By replacing d with dy and dj in the standard rate-distortion 
functions, we obtain the following expressions: 



R*tj(D)= inf UX-X + S) 

S:E[du(X : X+S)]<D 



R}(D) 



inf 

S:E[dj(X,X+S)]<D 



I(X;X + S) 



(43) 



(44) 



We note that the achievability and the converse proofs are 
straightforward extensions of the standard achievability and 
the converse proofs for regular rate distortion function. 

We next consider the distortion measures du and di and 
associated rate-distortion functions R^(D) and R}(D) when 
P -> 00. As P -> 00, E[d v {X n ,X n + S n )] < D implies 
E[d(X n ,X n + S n )} < D while E[X,S l ] -> for all i. 
Similarly, as -> 00, E[di(X n ,X n + S n )} < D implies 
E[d(X n ,X n + S n )} < D under the constraint that X t and Si 
are asymptotically independent for all i. Hence, as j3 — > 00, 
the distortion measures under consideration satisfy the respec- 
tive requirements of uncorrelatedness or independence, i.e., 
Ru{D) = R* V {D) and Rj(D) = R}(D). 

Hence, ( |39] l and ( |40] ) are indeed the information theoretical 
characterization of the limits of encoding a source with un- 
correlated and independent reconstruction error respectively. 



B. Gaussian Vector Source with MSE Distortion 

In this section, we examine a special case where the source 
is vector Gaussian and the distortion measure is squared error. 



We start with an auxiliary lemma without proof (see eg. 1 12 1 
for the proof). 

Lemma 1 ( | |12| ) Let S ~ fs and Sg ~ fs G ^ e random 
vectors in M, K with the same covariance matrix R$- If Sg ~ 
J\f(0, R s ) then 

E Sa pog(/ Sa (S))} - E s [log(/ SG (S))} (45) 

where Es G and E5 denote the expectations with respect to 
fs G and fs respectively. 
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Let us present a key lemma regarding the mutual informa- 
tion of two correlated random vectors constrained to have a 
fixed cross covariance matrix. 

Lemma 2 Let X ~ Af(0,R x ) and S G ~ M(0,R s ) be 
jointly Gaussian random vectors in M. . Let S £ VL K and 
S G have the same covariance matrix, Rs and the same cross 
covariance matrix with X, Rsx- Then, 



15 



Optimal Quantizer 

- Quantizer 1 
" Quantizer 2 

- Quantizer 3 
Uniform Dithered Q 



I(X,X + S) > I(X,X + S G ) 
with equality if and only if S ~ jV(0, Rs)- 



(46) 



Proof: Consider 7 = I(X, X + S) - I(X, X + S G ) 
Plugging the expressions, we obtain: 



7 = h{X\S G + X)- h(X\S + X) 



(47) 




Noting that h{X\S G + X) = h(S G \S G + X) and h(X\S- 
X) = h(S\S + X) and plugging Y = X + S and Y G = 
X + S G , we obtain: 



2.4 2.5 2.6 

Rate (bits) 

Fig. 3. Performance comparison in terms of SNR versus rate for fixed rate 
quantization. 



j = h(S G \Y G )-h(S\Y) 



(48) 



JJ{fsx( s >y) l °gfs\Y(s,y)-fsG,Y G (s,y)\ogf SGl Y G (s,y)}dsdy 

(49) 

Using Lemma [T] and the fact that the joint distribution fs G ,Y G 
is Gaussian: 

=jjfs,Y(s,y) [log f slY (s,y)-\og f SGl Y G {s,y)] dsdy 
a r \ f t 1 \ 1 fs\Y(s,y) 

my) / fs\Y{s,y)\og 



fs G \Y G (s,y) 

=T>(fs\Y,fs a \Y ) 

V > with equality if and only if S - Af(Q, R s ) 



(50) 

dsdy (51) 



(52) 



Next, we present our main result on this topic: 

Theorem 4 For a Gaussian vector source X eR K and MSE 
distortion d(x, x) = (x — x) T (x — x), the following holds: 



R T (D) = Ru(D) 



(53) 



Proof: Generally, Ri(D) > Ru(D), since independent 
reconstruction error is also uncorrelated. Note that the uncor- 
rected error constraint dictates Rsx — and the distortion 
constraint is Tr(Rs) = D. Lemma [5] states that under these 
constraints, for a Gaussian source, Gaussian reconstruction er- 
ror minimizes the mutual information between the source and 
the reconstruction, i.e., I(X G , Xq + S) achieves its minimum 
when S ~ A/"(0, Rs)- Then, X G and S are uncorrelated and 
jointly Gaussian and are, thereby, also independent. ■ 
We next pose the question: Are there cases where the best 
possible vector quantizer at asymptotically high dimension that 
renders the reconstruction error uncorrelated with the source, 
is necessarily a randomized one? The next corollary answers 
in the affirmative, as is proved by the Gaussian case. 

Corollary 1 For a Gaussian source, at asymptotically high 
quantizer dimension, the quantizer that achieves minimum 



distortion subject to the uncorrelated error constraint is nec- 
essarily a randomized one. 

Proof: From Theorem |4j the reconstruction error for the 
Gaussian source subject to the uncorrelated error constraint 
is independent of the source. No deterministic quantizer can 
render the quantization noise independent from the source by 
definition; hence, the optimal quantizer is a randomized one. 

■ 

Note that our results hold only asymptotically, it is still an 
open question whether or not they hold at finite dimensions. 
The numerical results in the next section support the thesis 
that randomized quantizers are better at finite dimensions. 

VI. Experimental Results 

In this section, we numerically compare the proposed quan- 
tizers to the conventional (uniform) dithered quantizer and 
to the optimal quantizer, for a standard unit variance scalar 
Gaussian source. We implemented the proposed quantizers 
by numerically calculating the derived integrals. For that 
purpose, we sampled the distribution on a uniform grid. We 
also imposed bounded support (—3 to 3) i.e., we numerically 
neglected the tails of the Gaussian. In this paper, we proposed 
three quantizers: 

Quantizer 1: Unconstrained randomized quantizer. This 
quantizer does not render the reconstruction error uncorrelated 
with the source. 

Quantizer 2: Constrained randomized quantizer which ren- 
ders the quantization error uncorrelated with the source. 

Quantizer 3: Constrained deterministic quantizer which 
renders the quantization error uncorrelated with the source. 

Figures 3 and 4 demonstrate the performance comparisons 
among quantizers for fixed and variable rates respectively. 
Note that for both fixed and variable rate, the optimal random- 
ized quantizer performs very close to the optimal quantizer. 
However, it does not provide the statistical benefits of the other 
quantizers. 
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Optimal Quantizer 

Quantizer 1 
" Quantizer 2 
■ Quantizer 3 

Uniform Dithered Q. 



0.12 





0.04 



0.02 - 



Fig. 4. Performance comparison in terms of SNR versus rate for variable Fig. 5. Correlation of the reconstruction error versus source correlation for 



rate quantization. 



fixed rate quantization at rate R = 2bits/sample. 



Note that for fixed rate, conventional (uniform) dithered 
quantization suffers significantly from the suboptimality of 
having equal quantization intervals irrespective of the rate 
region. However, at variable rate, the difference between the 
proposed and conventional dithered quantizer diminish at high 
rates, while at low rates the difference is quite significant. 
This is theoretically expected since at high rates, the optimal 
variable rate quantizer is very close to uniform, hence there 
is not much to gain from using a non-linear compressor- 
expander. 

For both fixed and variable rate, the constrained randomized 
quantizer outperforms its deterministic counterpart, while both 
of them perform significantly better than the conventional 
dithered quantizer. Both of the proposed quantizers render 
quantization error uncorrelated with the source with low 
performance degradation while the dithered uniform quantizer 
renders error independent of the source but (depending on the 
rate) at significant distortion penalty. 

An additional benefit of the proposed random quantizers 
pertains to the correlation of the reconstruction errors when 
correlated sources are quantized. The conventional dithered 
quantizer renders quantization error independent of the source 
hence, when two correlated sources are quantized with a 
dithered quantizer, the reconstruction errors are uncorrelated. 
For deterministic quantizers (Quantizer 3 and the optimal 
quantizer), the reconstruction error is a deterministic function 
of the source hence, intuitively randomized quantizers are 
expected to have lower reconstruction error correlation. Fig- 
ures 5 and 6 demonstrate the correlation of the reconstruction 
error for different values of source correlation for a bivari- 
ate Gaussian source for fixed and variable rate quantization 
respectively. These numerical results illustrate this intuitive 
conclusion: randomization is significantly useful in decreasing 
the correlations of reconstruction errors. Specifically, the con- 
strained randomized quantizer (Quantizer 2) yields extremely 
low error correlation, very close to that of the conventional 
dithered quantization. This property is particularly useful in 



0.1 



0.08 



0.02 



Optimal Quantizer 

Quantizer 1 

-e — Quantizer 2 
-H — Quantizer 3 
— Uniform Dithered Q. 




Fig. 6. Correlation of the reconstruction error versus source correlation for 
variable rate quantization at rate R = 1.4 bits/sample. 



practical applications such as image-video compression where 
white reconstruction error is preferred due to audio-visual 
considerations, see eg. deblocking filters commonly used in 
video coding [19]. Also note that, the unconstrained random- 
ized quantizer (Quantizer 1) significantly decreases the error 
correlation compared to the optimal quantizer, with negligible 
sacrifice in rate distortion performance. Hence, this statistical 
benefit of randomization comes with no significant penalty. 

Numerical comparisons show that the proposed quantization 
schemes can significantly impact the design of compression 
systems such as (7), (8| where quantization error is assumed 
to be uncorrelated with the source. Note that the constrained 
randomized quantization satisfies this assumption exactly and 
significantly outperforms the conventional dithered quantiza- 
tion, which has been presented in such prior work as the viable 
option to satisfy these assumptions. In fact, as an alternative 
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to the conventional dithered quantization that satisfies these 
assumptions at the considerable performance cost, we derived 
additional quantization schemes that satisfy those assumptions: 
constrained deterministic quantization and constrained nonuni- 
form random quantization. We also derived an unconstrained 
randomized quantizer, which performs almost as well as the 
optimal (deterministic) quantizer, yet offers perceptual benefits 
typical to dithered quantization. 

While it is difficult to prove, in general, the strict superi- 
ority of these new quantizers over the conventional dithered 
quantizer, we numerically show it, for both fixed and variable 
rate quantizers, in Figures 3 and 4. Moreover, the numerical 
results motivate a theoretical proposition: The optimal vector 
quantizer that renders the reconstruction error orthogonal 
to the source is necessarily randomized. While we proved 
this result at asymptotically high dimensions, it remains a 
conjecture at finite dimensions, based on the numerical results 
in this section. 

VII. Discussion 

In this paper, we proposed a nonuniform randomized quan- 
tizer where dithering is performed in the companded domain 
to circumvent the problem of matching the dither range to 
varying quantization intervals. The optimal compressor and 
expander mappings that minimize the mean square error are 
found via a novel numerical method. Also, we discovered 
the connections between the optimal quantizer and the one 
whose reconstruction error is constrained to be orthogonal 
to the source, for both deterministic and randomized quan- 
tization. The proposed constrained randomized quantization 
outperforms conventional dithered quantization and also the 
constrained deterministic quantizer proposed in this paper, 
while still satisfying the requirement that the reconstruction 
error be uncorrelated with the source. Moreover, the proposed 
randomized quantizers significantly reduce the correlations 
across reconstruction errors when correlated sources, i.e., 
sources with memory, are quantized. We also showed that 
at asymptotically high dimensions, the MSE optimal vector 
quantizer designed for a vector Gaussian source, which renders 
the reconstruction error uncorrelated with the source, must be 
a randomized quantizer. As future work, we will investigate 
the applicability of this result to a broader class of sources 
where random encoding is not merely a tool to derive rate- 
distortion bounds, but a necessary element in practical systems 
approaching such bounds. 
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